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To test the effectiveness of the Goal Oriented Teaching Exercise (GOTE). a 
six-day unit for measuring the effects of teaching strategies, four [unfor high school 
teachers received a teachers manual, information on instructional goals and subject 
content, and sample test questions all keyed to a content grid (formed by six content 
topics and two instructional goals— recall and application) for a teaching unit. Each 
teacher administered a pretest, also keyed to the grid, to students in two of his 
classes, and planned and taught lessons over a four-day period to one of these 
classes for recall and to the other for application. These lessons were observed and 
rated using the Observation Schedule and Record (OScAR) 5V. On the sixth day. a 
posttest (also keyed to the content arid) was administered to the students, and 
results were compared with those ot the pretest revealing differences in pupil 
learning by teachers and also bv goal. Audio tapes of the lessons were used to code 
psychological-social behavior of teachers (using OScAR). content covered, and type 
of instructional goal, and a visual display of interaction between these three types of 
teacher behavior was prepared. Teacner verbalism frequencies were tallied into 
content grid cells to show content coverage, and analyzed by goal, teacher, and 
observer visit revealing significant goal, visit, and observer, revealing 21 keys which 
reliably identified significant differences between teachers. (Appended are the 68 
items of the OScAR bV.) (SM) 
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GOAL ORIENTED TEACHING EXERCISE (G.O.T.E.): 
Methodology for measuring the effects of teaching strategies 



This pilot study reports the development of a device primarily de- 
signed for research in teaching effectiveness. Hopefully, it may also 
be developed into an instrument useful for exploration of teaching 
strategies, for evaluation of instruction, and for teacher- training. 



DESCRIPTION 



A Goal Oriented Teaching Exercise (G.O.T.E.) seeks to relate pat- 
terns of teacher behavior to pupil gains. A complete GOTE consists of 
(1) a four-day teaching unit, (2) specific objectives for pupils, (3) tests 
to measure pupil gains toward these specified objectives, (4) suggested 
teaching strategies, (5) instrumentation to describe teacher behavior, 
and (6) procedures for relating pupil gains to patterns of teacher be- 
havior. 

In this pilot study four teachers were asked to teach a unit on 
Air Pollution for specified content objectives. The unit content was 
organized along two dimensions? substantive content (e.g. causes, effects, 
etc.) and types of instructional goals (i.e. application and recall) to 
form twelve (12) content cells. Pupil gains were measured druing the 
teaching unit by equivalent forms of a test administered before and after 
the unit was taught. Teacher behavior was recorded three ways? by process, 
by substantive content, and by instructional goal. Each statement of the 
teacher was coded (1) using the OScAR 5V for information about social- 
psychological interaction, (2) by reference to the content cells of the 
total unit, and (3) by instructional goals. The pupil achievement data 
was then related to the measures of teacner behavior cell by cell. The 
data produced offered grounds for drawing inferences both about teaching 
strategies and their effects on pupils. 



RATIONALE 



The GOTE procedure builds upon four assumptions. The first assumpt- 
ion is that the teaching act is a complex multi-dimensional activity, and 
further, that teaching activities are interactions between cognitive ma- 
terials, goals of instruction, and teacher behavior. Secondly, research- 
ers can profitably invest in the development of more complex instrument- 
ation designed to describe these dimensions and their interaction. Pres- 
ently we have a series of first generation instruments for observing 
teacher behavior. These instruments must now be related to other kinds 
of measures to create new, more complex methodologies for describing and 
evaluating the complexity of the teaching act. The third assumption is 
that a description of the cognitive activities or content activities of 
the teacher is an important variable of the teaching act. This is suggest 
ed by Bellack (1966) in his study of teaching effectiveness when he points 



- 2 - 

out that it is important to describe or control the "content coverage" 
of the materials taught during his research upon teacher effectiveness. 
The fourth assumption is that control of the content can be developed 
through careful pre-planning wherein the content grid can become the 
basis for describing "teacher coverage" of the subject matter. It also 
can become the specification for diagnostic content measures of the stu- 
dent's achievement. 



DESCRIPTION OF GOTE MATERIALS 



GOTE materials were designed to accomplish three tasks. The first 
of these was to communicate the purposes and objectives of the study to 
the teachers who participated in the pilot study. The second task was 
to offer the participating teachers information and prescriptions which 
could guide them in carrying out these objectives. The third task was 
that of developing instruments that could measure the behavior of teach- 
ers and students. 

Efforts were made to develop materials which would allow replica- 
tion of the GOTE. This objective grew out of the hope that the GOTE 
methodology would grow and become a standardized procedure. The funct- 
ional effect of this value is illustrated in the care given to develop- 
ing a manual presenting the purposes and objectives of the GOTE study 
to the participating teachers. The same job could have been done with 
less effort via verbal communication with the small number of teachers 
involved in this initial effort. However, efforts were invented in 
developing a manual which could be used again. In all, there were 
five kinds of materials used and/or developed. They included: (1) 

a manual for teacher use, (2) a resource unit for the teaching activi- 
ties, (3) a statement of goal-i and sample test items, (4) four equiv- 
alent forms of the subject matter test, and (5) OScAR 5V , a teacher 
behavior observation instrument. 

The Teachers Manual presented the purposes of GOTE to the teachers 
who taught the units to be observed. The manual contained a discussion 
of procedures and the rationale of the GOTE activity. The concerns and 
the anxieties of the teachers were taken into account by a series of 
questions and answers which included, "How do I explain these activities 
to my class?" "Will this disrupt my class?" and "How can I get help in 
designing my teaching plan?" The objective of this manual was to com- 
municate with the participating teachers in a professional manner, and 
to secure their understanding and cooperation. 



The Resource Unit consisted of three parts. The first part pre- 
sented statements and illustrations of the two instructional goals of 
"recall" and "application". (Bloom 1956). The theory was briefly disr 
cussed. Sample test items were included to assist in drawing the dis- 
tinctions . 
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The second part of the resource unit specified the subject content 
to be taught in the space of four days. The subject chosen for this 
initial GOTE was that of "Air Pollution". This topic was chosen for three 
reasons* (1) it was a relatively new topic and students had not been ex- 
posed to it previously, (2) it could be justified in the context of a gen- 
eral science junior high curriculum as a current events topic, and (3) it 
had an appeal and interest for both students and teachers. 

The informational content to be taught concerning Air Pollution was 
organized into a "content grid". This content grid had twelve (12) cells 
divided by instructional goal and by topic. (Figure 1) 



FIGURE 1 

GOTE CONTENT GRID FOR AIR POLLUTION 
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These cells became the organizing structure for the uuTE content ma- 
terials. Teachers were given objectives and sample test questions keyed 
to these cells to help them target their instruction. Material and back- 
ground information was related to each cell. Events observed in the class- 
room were also related to these 12 cells. 

The third, or resource. phase of the unit included content objectives, 
reading materials for the teacher outlining the content information and 
suggested experiments for class use. A packet of materials was also in- 
cluded. It contained material aimed at the level of the pupils such as 
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visual aids and readings for class use. Again, all of these content 
materials were keyed to the cells of the content grid. 

Sample test questions were presented to each teacher. They served 
to illustrate the way their students would be tested for content knowledge 
and for educational goals. The objective was to help the teachers under- 
stand what they should teach. They were to be encouraged to do their best 
to get the best results possible from their children. 

Content tests were developed for the purpose of measuring pupil 
gains on a pre and post test basis. Four equivalent forms of twenty-six 
(26) items were created. Two or more items in each test were keyed to 
each cell of the afore mentioned "content grid". This test design yield- 
ed not only a single total score indicating how much a pupil has learned, 
but also diagnostic information about the kinds of things he has learned, 
so that gains made by a class during the exercise can be described both 
in terms of amount and pattern. It is conceivable that two classes might 
show the same amount of gain on a test (as measured by mean gains) al- 
though they have learned quite different things from teachers who have 
used quite different strategies. Such occurrences can have devastating 
effects on correlation between pupil gains and teaching behaviors. This 
test structure was designed to obviate this problem. 

It should be pointed out that the content tests were only checked 
for face validity. Efforts were not invested in rigorously establishing 
validity and reliability in this pilot effort. 

The OScAR 5V (Medley, 1968) was the instrument used to observe 
teacher social-emotional behavior during the teaching activities. The 
OScAR 5V is designed for recording information relevant to the affective 
and interpersonal interaction between teacher and pupil. The OScAR 5V 
has been described by its creator (Medley, 1968) "as a linear descendant 
of the Withall technique and a god-child (or step-child) . . .of the in- 
struments designed by H. H. Anderson (1945) and Flanders (I960)." The 
OScAR instrument has evolved through several stages as suggested by the 
numerals 5V. The present instrument contains eighteen (18) separate cat- 
egories - four for pupil utterances and fourteen for teacher utterances. 
Since six of the teacher categories are dual-purpose, however, we can 
say that there are four categories for classifying pupils and twenty for 
teacher behaviors. The number of discinct types of events which may be 
recognized and recorded, however, is considerably greater than twenty- 
four because of the many ways in which entries and exits may be combined 
in the same event. The manual for the observational instrument lists 
sixty-eight different events -- thirteen kinds of statements and fifty- 
five kinds of interchanges - which form the basis of inference about class- 
room environment made from OScAR records. 



SAMPLE 



Four teachers agreed to participate in the pilot GOTE study. 



They 



were all junior high school science teachers at the same junior high 
school teaching children of generally comparable ability. In consul- 
tation with the investigators, each teacher selected two classes which 
seemed to be comparable in ability. The classes were either on a 
seventh or eighth grade level. It was assumed that the effect of dif- 
ferences between classes wouid be reduced by the use of the pre and post 
test procedure to measure student gains. 



PROCEDURE 



The initial step in the procedure was the explanation of the pro- 
ject to the participating teachers. Members of the staff met with teach- 
ers several times and discussed the purposes and objectives of the study 
and outlined the procedures to be followed. The teachers were asked to 
teach one of their classes for "recall" and one of their classes for "ap- 
plication”. The assumption was that experienced teachers could differ- 
entiate their teaching style if they knew they were to teach for diff- 
erent kinds of objectives. The teachers prepared lessons for four days 
of teaching drawing upon the materials provided by the investigators, 
including the subject matter grids. Help was offered by the investiga- 
tors in discussing lesson plans, but it was made clear that the teachers 
were to take the leadership in developing the procedures and approach. 

The GOTE unit covered a period of six days. The first day consisted 
of an explanation to the students of what was happening and a pre-test. 
This was followed by four days of teaching during which time staff of the 
GOTE project observed the behavior of teachers. The sixth day consisted 
of a post-test after which questions were answered and discussions with 
the students were held about the overall experience. 

Observational procedures during the pilot test were of three kinds. 
Video tapes were made of each teacher in each of the eight classes for one 
day for a total of eight tapes. Audio- tapes were made of all lessons of 
all teachers. In addition, two observers were in the room of the teacher 
during all teaching time. The observers used the OScAR 5V form and noted 
impressions as the teaching unit proceeded. 

There were several problems in the procedure. There was no uniform- 
ity among teachers as to whether they would count the work done as cred- 
it toward report card marks. Another area of unanticipated confusion lay 
in the use of homework. One teacher began setting expectations for home- 
work that were quite different from the expectations of the other teach- 
ers. Either procedure would have produced new and unexpected variables 
in the procedure. These issues were dealt with on a post-hoc emergency 
basis and guidelines were set up to insure a degree of uniformity in the 
eight classes. 

The sound observations and the video observations were successful. 

The quality of the student response on the audio and video tapes was 
reasonably good. The observers using OScAR 5V had little difficulty in 
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keeping track of the behavior of the teacher and in interpreting the 
behavior on the basis of a consistent set of ground rules. The teachers 
seemed interested in the project and were pleased to participate from a 
professional point of view. They were also pleased with the subject 
matter and with the response of their students. 

After the teaching phase of the GOTE was completed and the data was 
collected, the audio- tapes were analyzed. Each verbal statement was again 
classified on the basis of psychological process using OScAR 5V; in addi- 
tion, it was also classified according to instructional goal and content 
concept. This information was combined and prepared for optical scanning 
for the purpose of statistical analysis. (See Page No. 7) 



KINDS OF DATA COLLECTED 



In all, four kinds of data were collected: (1) recordings, (2) 

measures of pupil gains, (3) measures of content coverage, and (4) ob- 
servations of teacher social-emotional interaction. The treatment and 
display of each type of data is as follows: 

The video tapes were simply stored. At times it was valuable to 
view selected portions of the teaching activities of a teacher. The 
audio tapes were used to code both the OScAR behavior and the content 
cells. The great value of these recordings lies in the opportunity they 
offer to those who would like to examine the teaching activities of the 
GOTE with some other mode of analysis. 

The measures of pupil gains were analyzed in a simple linear mode. 
Average gains were computed for individual test items per individual pu- 
pil within a class. An average gain of .14 per item meant that the pupils 
in the class gained an average of 3.64 points on a twenty-six item test. 
Results were reported by class. This provided information about pupil 
learning by teachers and also by goal. The data was further analyzed to 
reveal differences in kinds and patterns of learning by contrasting groups 
of cells, e.g. control cells vs. effects. A value of .14 here would mean 
gains in content cells exceeded gains in effects cells by 3.64 points. 
Contrast of cells was established, e.g. control of Air Pollution vs. ef- 
fects of Air Pollution. These contrasts provided diagnostic information 
about pupil learning. (Figure 2) 



FIGURE 2 
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PROCEDURE FOR CODING GOTE DATA 



Three kinds of data were collected concerning teacher behavior; psychological- 
social behavior, content coverage, and instructional objectives. This information 
was prepared for optical scanning in such a way that there was a visual display of 
the interaction between the three types of data over time. 58 
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Inspection of the display reveals the simultaneous interaction of the three 
kinds of data. For example, in column IV 3, the teacher raises a problem structuring 
question (OScAR data) concerned with the cause of Air Pollution (Content data) which 
is oriented toward recall (Instructional Objective data). Again, in Column VI3, the 
pupil offers substantive information concerning the application of information about 
the sources of Air Pollution to which the teacher gives positive feedback. 
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Data about content coverage by the teacher was displayed and an- 
alyzed in two ways. First, the frequencies were tallied into the con- 
tent cells. These cells were analyzed for coverage by inspection. Sev- 
eral indexes were established by inspection. For example, the number of 
cempty cells was noted as an indication of content coverage 

Secondly, the frequencies of teacher verbalisms in the content cells 
were subjected to a three-way analysis of variance, by teacher, goal and 
visit, and their interactions. 

The results of this analysis indicate that there were significant 
differences in the content coverage of the informational content. Con- 
tent coverage is a major variable in measurements of teaching effective- 
ness. (Figure 3) 



FIGURE 3 
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Data concerning t eacher social-psychological behavior was collected using OScAR 
5V. The sixty-eight (68) possible scores were tabulated and analyzed for variance 
(ANOVA) considering teacher, goal, visit, recorder, and their interactions. Twenty- 
one (21) keys reliably identified significant differences between teachers. These 
items are as follows: 



Informing statement ++ 
Directing statement ++ 
Rebuking statement ++ 

Non- substantive question ++ 



Procedural question - positive 44- 
Pupil non- substantive - approved 44- 
Pupil question - not evaluated 44- 

Pupil question - approved 44- 

Pupil statement - approved 4- 
Pupil response 4- 

Pupil response - not evaluated 44- 

Pupil response - approved 4- 

Pupil response - acknowledged 4* 

Problem structuring statement 44- 
Convergent interchange - not evaluated 44- 
Convergent interchange - supported 44- 
Convergent interchange - acknowledged 44- 
Convergent interchange - neutrally rejected 4- 
Elaborating question (1) - not answered 44- 
Elaborating interchange (2) acknowledged 44- 

4- significance at the .05 level 
44- significance at the .01 level 

Two items distinguished by visit - thirteen items distinguished teacher 
by goal - twenty-four items distinguished teacher by visit. 



POSSIBLE INFERENCES 

For purposes of this paper, selected facets of each kind of data are 
high-lighted. Other possible hypotheses are also suggested. The objective 
is to suggest the range and productivity of the GOTE methodology. 

Measures of Student Gains ; Inspection of measures of pupil gains reveals 
that students in the classes of Teacher #2 learned substantially more than 
the students of the other teachers. This was particularly true when Teacher 
#2 taught for the instructional goal of application. (Figure 4). The av- 
erage gain of the children in his class was more than seven times greater 
than the average gain of the class which learned the least. (Figure 2). 

It might be said, then, that Teacher #2 was the most effective teacher of 
the four, particularly when he was teaching for application, and that Teacher 
#1 was the least effective teacher. 

However, inspection of the contrasts of pupil gain scores suggests that 
there was a difference in the learning of the students by goal and by con- 
tent cell. It can be hypothesized that Teacher #4 had a teaching style that 
was more effective in teaching application than recall. Indeed, he was more ef- 
fective in this sense than any of the other three teachers, all of whose pu- 
pils gained more on recall items than on application ones . Also note that 
when Teacher #4 was supposed to stress recall as his objective, his pupils 
gained more in application (.09) than when application was his supposed goal 
(.03). Moreover, his classes learned more about Techniques of Measurement 
as against Facts of Air Pollution and more about the Control of Air Pollution 
as contrasted with the Effects of Air Pollution. 



AVERAGE AMOUNT OF INDIVIDUAL STUDENT GAIN 
WITHIN CLASSES PER ITEM 



- 10 - 



FIGURE 4 

GOTE ACHIEVEMENT MEASURES 
Gains of Pupils X Classes X Teachers 
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Student gains were measured pre and post GOTE using four equiv- 
alent test forms. Results are reported as average individual 
student gains within class per item. (The highest gain of .22 
per item is equivalent to a gain of 5.7 on a 26 item test.) 
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Measures of Content Coverage ; It is interesting to trace the content cov- 
erage of Teacher if 2 (hypothesized as the most effective teacher). He dif- 
fers from the other teachers in the number of content cells covered. When 
teaching for the goal of application , the other three teachers has a mean of 
27 content cells with no verbal entries. Teacher if 2 had only 20 vacant cells. 
He covered the material more effectively! 

The analysis of variance of the content cells revealed differences and 
similarities in the way the teacher covered the subject matter. This data 
provides rich ground for inferences about teacher content strategies, (Teacher 
by visit interactions.) 

Measures of Teacher Social-Psychological Behavior : Teacher if 2 again stands 

out. He exhibits a different pattern of behavior than the other teachers as 
a group. The differences between Teacher if 2 and Teacher ifl when teaching 
for the goal of application are most interesting. In a sense, they are a 
comparison of the behavior of the most effective teacher with the behavior 
of the least effective teacher. For purposes of simple contrast, the be- 
haviors on which these teachers loaded both the least and the most are dis- 
played. This provides a glimpse of the most prominent behavior characterist- 
ics of these two teachers. 

TEACHER if 2 ON GOAL OF APPLICATION (Hypothesized as the most effective teacher) 
Scored highest of all teachers on the following items: 

Informing statements 
Pupil non-substantive - approved 
Pupil questions - not evaluated 
Pupil questions - approved 
Pupil statements - approved 
Pupil responses - acknowledged 
Convergent interchanges - supported 
Elaborating questions - not answered 

TEACHER ifl ON GOAL OF APPLICATION (Hypothesized as the least effective teacher) 
Scored highest of all teachers on the following items: 

Directing statements 
Rebuking statements 
Non-substantive statements 
Pupil responses - not evaluated 
Problem structuring statements 
Convergent interchanges - not evaluated 
Convergent interchanges - acknowledged 
Convergent interchanges - neutrally rejected 

Scored lowest of all teachers on the following items; 

Informing statements 
Procedural questions - positive 
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There are many other ways of analyzing the data produced by this 
pilot GOTE. The three kinds of data have yet to be combined to examine 
the congruence of teacher behavior) content coverage) and student learn"" 
ing. Several other kinds of indexes can be developed. An example being 
examined is that of "individual teacher strategies" which is defined as 
the frequency pattern by visit of items (which is reflected in the inter- 
action "teacher by visit" in the analysis of variance). 

GOTE data output provides a basis for analyses which combine quant- 
itative measures with clinical or case study detail. These possibilities 
will be explored further in a forthcoming publication. 



ERIC 
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